Upper and Lower Bounds on the Cost of a Map-Reduce Computation

نویسندگان

  • Foto N. Afrati
  • Anish Das Sarma
  • Semih Salihoglu
  • Jeffrey D. Ullman
چکیده

In this paper we study the tradeoff between parallelism and communication cost in a map-reduce computation. For any problem that is not “embarrassingly parallel,” the finer we partition the work of the reducers so that more parallelism can be extracted, the greater will be the total communication between mappers and reducers. We introduce a model of problems that can be solved in a single round of mapreduce computation. This model enables a generic recipe for discovering lower bounds on communication cost as a function of the maximum number of inputs that can be assigned to one reducer. We use the model to analyze the tradeoff for three problems: finding pairs of strings at Hamming distance d, finding triangles and other patterns in a larger graph, and matrix multiplication. For finding strings of Hamming distance 1, we have upper and lower bounds that match exactly. For triangles and many other graphs, we have upper and lower bounds that are the same to within a constant factor. For the problem of matrix multiplication, we have matching upper and lower bounds for one-round map-reduce algorithms. We are also able to explore tworound map-reduce algorithms for matrix multiplication and show that these never have more communication, for a given reducer size, than the best one-round algorithm, and often have significantly less.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

A Survey on Stability Measure of Networks

In this paper we discuss about tenacity and its properties in stability calculation. We indicate relationships between tenacity and connectivity, tenacity and binding number, tenacity and toughness. We also give good lower and upper bounds for tenacity.

متن کامل

Estimating ‎U‎pper and Lower Bounds For Industry Efficiency With Unknown ‎Technology‎

With a brief review of the studies on the industry in Data Envelopment Analysis (DEA) framework, the present paper proposes inner and outer technologies when only some basic information is available about the technology. Furthermore, applying Linear Programming techniques, it also determines lower and upper bounds for directional distance function (DDF) measure, overall and allocative efficienc...

متن کامل

Upper and lower bounds of symmetric division deg index

Symmetric Division Deg index is one of the 148 discrete Adriatic indices that showed good predictive properties on the testing sets provided by International Academy of Mathematical Chemistry. Symmetric Division Deg index is defined by $$ SDD(G) = sumE left( frac{min{d_u,d_v}}{max{d_u,d_v}} + frac{max{d_u,d_v}}{min{d_u,d_v}} right), $$ where $d_i$ is the degree of vertex $i$ in graph $G$. In th...

متن کامل

Upper and lower bounds for numerical radii of block shifts

For an n-by-n complex matrix A in a block form with the (possibly) nonzero blocks only on the diagonal above the main one, we consider two other matrices whose nonzero entries are along the diagonal above the main one and consist of the norms or minimum moduli of the diagonal blocks of A. In this paper, we obtain two inequalities relating the numeical radii of these matrices and also determine ...

متن کامل

Capacity Inverse Minimum Cost Flow Problem under the Weighted Hamming Distances

Given an instance of the minimum cost flow problem, a version of the corresponding inverse problem, called the capacity inverse problem, is to modify the upper and lower bounds on arc flows as little as possible so that a given feasible flow becomes optimal to the modified minimum cost flow problem. The modifications can be measured by different distances. In this article, we consider the capac...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:
  • PVLDB

دوره 6  شماره 

صفحات  -

تاریخ انتشار 2013